{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpJd3dlOCStH"
      },
      "source": [
        "\u003ca href=\"https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/tutorials/2_processor_group.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hMqWDc_m6rUC"
      },
      "source": [
        "\n",
        "##### Copyright 2021 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VNhgka4UKNjf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFIqwYGbZ-df"
      },
      "source": [
        "# DDSP ProcessorGroup\n",
        "\n",
        "This notebook demonstrates the use of a `ProcessorGroup()` as an alternative to stringing signal `Processors()` together in python. \n",
        "\n",
        "The main advantage of using a ProcessorGroup is that the entire signal processing chain can be specified in a `.gin` file, as a Directed Acyclic Graph (DAG), removing the need to write code in python for every configuration of processors.\n",
        "\n",
        "\n",
        "In this tutorial we're going to synthesize some audio from these example controls in three different ways.\n",
        "\n",
        "* With processors and python control flow\n",
        "* With a ProcessorGroup DAG (via list)\n",
        "* With a ProcessorGroup DAG (via gin)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "bW54esgl1EZ1"
      },
      "outputs": [],
      "source": [
        "#@title Install and import dependencies\n",
        "\n",
        "%tensorflow_version 2.x\n",
        "!pip install -qU ddsp\n",
        "\n",
        "# Ignore a bunch of deprecation warnings\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import ddsp\n",
        "import ddsp.training\n",
        "from ddsp.colab.colab_utils import play, specplot, DEFAULT_SAMPLE_RATE\n",
        "import gin\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "sample_rate = DEFAULT_SAMPLE_RATE  # 16000"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ixvOEZy3lnZ4"
      },
      "source": [
        "# Example processor inputs\n",
        "\n",
        "Some signals to be used in the rest of the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wj6SWcZYZRD4"
      },
      "outputs": [],
      "source": [
        "# Constants\n",
        "n_frames = 1000\n",
        "hop_size = 64\n",
        "n_samples = n_frames * hop_size\n",
        "\n",
        "\n",
        "#### Harmonic controls\n",
        "# Amplitude [batch, n_frames, 1].\n",
        "amps = np.linspace(0.5, -5.0, n_frames)[np.newaxis, :, np.newaxis]\n",
        "\n",
        "# Harmonic Distribution [batch, n_frames, n_harmonics].\n",
        "n_harmonics = 20\n",
        "harmonic_distribution = np.ones([n_frames, 1]) * np.linspace(1.0, -1.0, n_harmonics)[np.newaxis, :]\n",
        "for i in range(n_harmonics):\n",
        "  harmonic_distribution[:, i] = 1.0 - np.linspace(i * 0.09, 2.0, 1000)\n",
        "  if i % 2 != 0:\n",
        "    harmonic_distribution[:, i] = -3\n",
        "harmonic_distribution = harmonic_distribution[np.newaxis, :, :]\n",
        "\n",
        "# Fundamental frequency in Hz [batch, n_frames, 1].\n",
        "f0_hz = np.linspace(300.0, 200.0, n_frames)[np.newaxis, :, np.newaxis]\n",
        "\n",
        "\n",
        "### Filtered Noise controls\n",
        "# Magnitudes [batch, n_frames, n_magnitudes].\n",
        "n_filter_banks = 20\n",
        "magnitudes = np.linspace(-1.0, -4.0, n_filter_banks)[np.newaxis, np.newaxis, :]\n",
        "magnitudes = magnitudes + amps \n",
        "\n",
        "\n",
        "### Reverb controls\n",
        "n_fade_in = 16 * 10\n",
        "ir_size = int(sample_rate * 2)\n",
        "n_fade_out = ir_size - n_fade_in\n",
        "\n",
        "ir = 0.01 * np.random.randn(ir_size)\n",
        "ir[:n_fade_in] *= np.linspace(0.0, 1.0, n_fade_in)\n",
        "ir[n_fade_in:] *= np.exp(np.linspace(0.0, -5.0, n_fade_out))\n",
        "ir = ir[np.newaxis, :]\n",
        "\n",
        "\n",
        "inputs = {\n",
        "    'amps': amps,\n",
        "    'harmonic_distribution': harmonic_distribution,\n",
        "    'f0_hz': f0_hz,\n",
        "    'magnitudes': magnitudes,\n",
        "    'ir': ir,\n",
        "}\n",
        "inputs = {k: v.astype(np.float32) for k, v in inputs.items()}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lD_Kz37ZqSAv"
      },
      "outputs": [],
      "source": [
        "# Plot the inputs\n",
        "time = np.linspace(0, n_samples / sample_rate, n_frames)\n",
        "\n",
        "plt.figure(figsize=(18, 8))\n",
        "plt.subplot(231)\n",
        "plt.plot(time, amps[0, :, 0])\n",
        "plt.xticks([0, 1, 2, 3, 4])\n",
        "plt.title('Amp Input')\n",
        "\n",
        "plt.subplot(232)\n",
        "plt.plot(time, harmonic_distribution[0])\n",
        "plt.xticks([0, 1, 2, 3, 4])\n",
        "plt.title('Harmonic Input')\n",
        "\n",
        "plt.subplot(233)\n",
        "plt.plot(time, f0_hz[0, :, 0])\n",
        "plt.xticks([0, 1, 2, 3, 4])\n",
        "plt.title('Fundamental Frequency')\n",
        "\n",
        "plt.subplot(234)\n",
        "plt.plot(ir[0])\n",
        "plt.title('Impulse Response')\n",
        "\n",
        "plt.subplot(235)\n",
        "plt.plot(time, magnitudes[0])\n",
        "plt.xticks([0, 1, 2, 3, 4])\n",
        "_ = plt.title('Noise Magnitudes')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7j5XCUJkK9WZ"
      },
      "source": [
        "# Processors\n",
        "\n",
        "You can generate signal by stringing Processors together in python, as you would with any other differentiable modules."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mhe014IirnHi"
      },
      "outputs": [],
      "source": [
        "harmonic = ddsp.synths.Harmonic(n_samples=n_samples)\n",
        "noise = ddsp.synths.FilteredNoise(n_samples=n_samples, initial_bias=0)\n",
        "reverb = ddsp.effects.Reverb()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xf7Vc3UtNQ87"
      },
      "outputs": [],
      "source": [
        "# Python signal processor chain\n",
        "audio_harmonic = harmonic(inputs['amps'],\n",
        "                          inputs['harmonic_distribution'],\n",
        "                          inputs['f0_hz'])\n",
        "audio_noise = noise(inputs['magnitudes'])\n",
        "audio_dry = audio_harmonic + audio_noise\n",
        "audio_out = reverb(inputs['ir'], audio_dry)\n",
        "\n",
        "# Listen\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0oJJunbPOZ3R"
      },
      "source": [
        "# ProcessorGroup\n",
        "\n",
        "A ProcessorGroup is a Directed Acyclic Graph (DAG) of Processors.\n",
        "\n",
        "You can specify the DAG as a list of tuples `dag = [(processor, ['input1', 'input2', ...]), ...]`, where each tuple is a pair of processor and that processor's inputs respectively.\n",
        "\n",
        "The output signal of any processor can be referenced as an input to a different processor by the string `'processor_name/signal'` where processor_name is the name of the processor at construction.\n",
        "\n",
        "The ProcessorGroup takes a dictionary of inputs, whose keys are referenced as inputs in the DAG.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mFEIVLa1INTJ"
      },
      "outputs": [],
      "source": [
        "print(inputs.keys())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qXSRq5nXFVnd"
      },
      "outputs": [],
      "source": [
        "harmonic = ddsp.synths.Harmonic(n_samples=n_samples, name='harmonic')\n",
        "noise = ddsp.synths.FilteredNoise(n_samples=n_samples, name='noise', initial_bias=0.0)\n",
        "reverb = ddsp.effects.Reverb(name='reverb')\n",
        "add = ddsp.processors.Add(name='add')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vcvo7T7mFVnk"
      },
      "outputs": [],
      "source": [
        "# Processor group DAG\n",
        "dag = [\n",
        "  (harmonic, ['amps', 'harmonic_distribution', 'f0_hz']),\n",
        "  (noise, ['magnitudes']),\n",
        "  (add, ['harmonic/signal', 'noise/signal']),\n",
        "  (reverb, ['ir', 'add/signal'])\n",
        "]\n",
        "\n",
        "processor_group = ddsp.processors.ProcessorGroup(dag=dag)\n",
        "audio_out = processor_group(inputs)\n",
        "\n",
        "# Listen\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FiXscfWu09Jt"
      },
      "outputs": [],
      "source": [
        "processor_group.get_controls(inputs)['out']['signal']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cFKnBKpbYzZL"
      },
      "source": [
        "The processor group also offers all the intermediate signals and control tensors for inspection."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "exsKvj_aY_se"
      },
      "outputs": [],
      "source": [
        "outputs = processor_group.get_controls(inputs)\n",
        "\n",
        "np.set_printoptions(precision=1, threshold=0, edgeitems=1)\n",
        "for k, v in outputs.items():\n",
        "  print(\"'{}':\".format(k), v)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OhaovQCva13U"
      },
      "outputs": [],
      "source": [
        "noise_audio = outputs['noise']['signal']\n",
        "play(noise_audio)\n",
        "specplot(noise_audio)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UZnW__RBbLo1"
      },
      "outputs": [],
      "source": [
        "harmonic_distribution_np = outputs['harmonic']['controls']['harmonic_distribution']\n",
        "_ = plt.matshow(np.rot90(harmonic_distribution_np[0, :, :]), aspect='auto')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_7ksFXMOnxY"
      },
      "source": [
        "# Configuration with Gin\n",
        "\n",
        "The main advantage of a ProcessorGroup is that it can be defined with Gin, allowing flexible configurations without having to write new python code for every DAG of processors.\n",
        "\n",
        "In the example below we pretend we have an external file written, which we treat here as a string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qHsqd_lJOnxZ"
      },
      "outputs": [],
      "source": [
        "gin_file_string = \"\"\"\n",
        "import ddsp\n",
        "\n",
        "processors.ProcessorGroup.dag = [\n",
        "  (@Harmonic(), ['amps', 'harmonic_distribution', 'f0_hz']),\n",
        "  (@FilteredNoise(), ['magnitudes']),\n",
        "  (@Add(), ['noise/signal', 'harmonic/signal']),\n",
        "  (@Reverb(), ['ir', 'add/signal'])\n",
        "]\n",
        "\n",
        "Harmonic.name = 'harmonic'\n",
        "FilteredNoise.name = 'noise'\n",
        "processors.Add.name = 'add'\n",
        "\n",
        "Harmonic.n_samples = 64000\n",
        "FilteredNoise.n_samples = 64000\n",
        "FilteredNoise.initial_bias = 0.0\n",
        "\"\"\"\n",
        "with gin.unlock_config():\n",
        "  gin.parse_config(gin_file_string)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bnTkWeoeLSfv"
      },
      "source": [
        "Now, after parsing the gin file, the ProcessorGroup will be configured on construction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BpbkMwMlKVLH"
      },
      "outputs": [],
      "source": [
        "processor_group = ddsp.processors.ProcessorGroup()\n",
        "audio_out = processor_group(inputs)\n",
        "\n",
        "# Listen\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "hMqWDc_m6rUC",
        "ZFIqwYGbZ-df",
        "ixvOEZy3lnZ4",
        "7j5XCUJkK9WZ",
        "0oJJunbPOZ3R",
        "T_7ksFXMOnxY"
      ],
      "last_runtime": {},
      "name": "2_processor_group.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
